Tell a card awaiting its first index pass apart from one that is missing by habdelra · Pull Request #5929 · cardstack/boxel

habdelra · 2026-08-28T16:02:54Z

A card's source file lands on the realm's file system before the indexing pass that makes it servable. card+json — the read every top-level card load goes through — is served from the index, so during that window the read comes back 404. That 404 is shaped exactly like the one for a card that was never there, and the host treats it the same way: the card you just created or uploaded shows up as a missing card, complete with a runtime-error banner, until something forces a reload.

Two things narrow the window but neither closes it. The replica that handled the write drains its own in-flight indexing before serving a card+json read, so a single-replica setup usually gets a slow 200 instead of a fast 404 — but a read served by any other replica has no handle on that write. And linksTo targets resolve through card+source, which reads the file rather than the index, so linked cards were never affected. What's left is the top-level read, which is precisely the one that runs right after you create or upload something.

The realm now looks at the source file before answering. When the index has no row but the .json is on disk and holds a card document, the 404 carries an awaitingIndex marker: the reference is sound, indexing just hasn't caught up. A path with no source file behind it keeps the plain 404 it always had, so "not there" and "not there yet" stay distinct.

The host reads that marker off the card error and renders a placeholder instead of the error chrome. Nothing else changes: the store already keeps the error keyed by the card's id and already reloads it when the realm broadcasts the index event for that id, so the placeholder resolves into the real card on its own, with no reload and no user action. Because the branch sits in the CardError component, the surfaces that reach it — stack items, code-mode preview, the playground — pick it up together. Host mode is not among them: it branches on status === 404 ahead of CardError and shows its own "This page could not be found" screen. Leaving it there is deliberate rather than an oversight — a published page holds no realm subscription (subscribeToRealm returns early when host mode is active), so a placeholder shown there would have no index event to resolve it. Giving host mode this state needs a resolution path of its own.

A card that is already running in this tab is never covered by the placeholder. A newly created instance is live in the store under its local id, and editable there, long before the realm has indexed it; the store declines to record an awaiting-index error over a running card, hands the running card back to a cache-bypassing read that meets one, and leaves the instance untouched when a reload meets one. The placeholder is for a card this tab has never held.

Before / after

The same card, mid-index, before and after:

Testing

packages/realm-server/tests/card-endpoints-test.ts — dropping a card's index row while leaving its .json on disk produces a 404 marked awaitingIndex, on both the plain and the conditional-GET path; a path with no source file behind it does not.
packages/host/tests/integration/components/card-error-awaiting-index-test.gts — a marked error renders the placeholder and offers no error detail; an unmarked 404 still renders the card error.
packages/host/tests/integration/store-test.gts — end to end: a card written to the realm without an indexing pass renders as being prepared, and swaps itself for the real card once the realm indexes it.

🤖 Generated with Claude Code

A write lands on the realm's file system before it is indexed, and card+json is served from the index. In that window a read for the new card gets a 404 that is indistinguishable from "this card does not exist", so the host commits to the missing-card treatment for a card that is on its way. The realm now checks the source file before answering: when the `.json` is on disk and holds a card document, the 404 carries an `awaitingIndex` marker. The host renders a "Preparing this card" placeholder for such an error, and the store's existing invalidation-driven reload swaps in the real card when the realm broadcasts the index event. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 63b1722e3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-08-28T16:08:09Z

Preview deployments

Host staging preview

Host production preview

Host Test Results

1 files ± 0 1 suites ±0 2h 29m 32s ⏱️ +52s
4 627 tests +24 4 613 ✅ +24 14 💤 ±0 0 ❌ ±0
4 642 runs +24 4 628 ✅ +24 14 💤 ±0 0 ❌ ±0

Results for commit 90991a6. ± Comparison against earlier commit 42dece3.

Realm Server Test Results

1 files ± 0 1 suites ±0 22m 8s ⏱️ + 3m 40s
2 334 tests - 21 2 334 ✅ - 21 0 💤 ±0 0 ❌ ±0
2 417 runs - 21 2 417 ✅ - 21 0 💤 ±0 0 ❌ ±0

Results for commit 90991a6. ± Comparison against earlier commit 42dece3.

The awaiting-index marker told a caller to wait, so it has to be reserved for sources the indexer will actually give an instance row: a `.json` whose `data` is a single card resource, at a path the indexer does not ignore. A collection document — which `isCardDocumentString` also accepts — and an ignored path never acquire a row, so those stay a plain 404. An invalidation naming a card whose first read is still in flight also no longer goes to waste. The store had nothing to reload at that point and dropped the event, and the awaiting-index placeholder that read went on to install would then be stale with no further event coming for it. The store now waits out the in-flight read and reloads if it left a placeholder behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The realm broadcasts its own index event when it indexes the file, and matrix hands that over after the read has settled — where the ordinary error-reload path picks it up. The test now delivers exactly one event, inside the window, so it fails when the in-flight check is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… index A from-scratch pass at realm startup announces itself with a bare `full` index event and no per-card invalidations, so it can be the only word a card being held as awaiting-index gets that its row now exists. The store re-reads its awaiting-index placeholders on that event. The header keeps the `error-header` class consumers style and only adds a `pending` modifier, and names its own text colour: the colour it was inheriting comes from the realm's own colour, which says nothing about the grey the pending state paints — white on near-white for any dark realm. Its icon is the loading-indicator component rather than the raw glyph, so it actually turns, and the placeholder is a polite live region so the card arriving is announced and not just drawn. The deferred in-flight reload no longer counts itself in the realm-event telemetry, since whether it reloads depends on what the read settles into. Adds an acceptance test covering both sides of the wait through the real routing and stack-item path, and an integration test for the full-index sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A 404 on reload means the index no longer has a row, normally because the file was deleted — so the store evicts the instance and rewrites every consumer's link to a not-found sentinel. When the realm reports the source is still there and only awaiting indexing, none of that is true, and the card goes to the awaiting-index placeholder like any other failed reload. Also trims the placeholder's second sentence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

habdelra · 2026-08-29T12:49:19Z

There is a major regression here. in code mode when I use the New+ button to create a card instance (just a base card), I have to wait for indexing to complete before I can start editing the card. THIS IS WRONG. the store should provide a running card with a local ID that is 100% renderable in the card playground. we should not have to round trip thru the server before we can start working with this card. your changes made it such that indexing is now GATING my ability to interact with a newly created card. the whole idea where was that the creating sentinel was a sidecar in the store such that if we went to FETCH the card we would get a creating message instead of a 404. but in the case we just are interacting with the card directly that is already running in the store we should not be prevented from doing so. is that clear? never step in front of an actual running card in the store--only wait when there is a fetch for a newly created card. and you should be able to use a local ID to correlate just like the store does for newly created cards.

A card created in this tab is live under its local id and editable there long before the realm has indexed it. The realm reporting that it has not caught up is a statement about the index, not about that instance — but recording it as an error made `peekError` report it, and every render site reads that to decide whether to show a placeholder, so a card the user was working in was replaced by one and its autosave was detached. The store no longer records an awaiting-index error over a running card, hands the running card back when a cache-bypassing read meets one, and leaves the instance untouched when a reload does. `getCard` correlates a remote URL back to a locally-created instance, so this holds from the moment the server assigns an id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

habdelra · 2026-08-29T21:03:50Z

[Claude Code 🤖] Confirmed and fixed in e46aa51. You were right about where the sentinel belongs — it was leaking out of the fetch path and into the store's identity map.

The mechanism: an awaiting-index 404 was recorded through setIdentityContext like any other card error, so store.peekError(id) reported it. Every render site (stack item, playground, code-mode preview) reads peekError to decide whether to stand something in front of the card, so a running instance got covered even though peek still returned it. Worse, in reloadTask the error replaced the instance as the reload result, which tripped stopAutoSaving on it — that is the part that actually gated editing rather than just hiding the card.

Three changes, all keyed on "is there a running instance for this id":

setIdentityContext does not record an awaiting-index error when store.getCard(id) finds a live instance. That is the chokepoint every path stores errors through, so nothing downstream can reintroduce it. getCard already correlates a remote URL back to a locally-created instance — including the last-segment-to-local-id correlation — so it holds from the moment the server assigns the id, not just after the resolver learns it.
A cache-bypassing read that meets an awaiting-index 404 returns the running card instead of the error.
A reload that meets one leaves the instance exactly as it is, autosave included, and waits for the index event.

The placeholder now only appears where it was meant to: a fetch for a card this tab has never had.

packages/host/tests/integration/store-test.gts gains a test that creates a card, holds a running instance, and asserts that an awaiting-index 404 arriving for it leaves peek on the instance, peekError empty, and autosave attached. It fails on all three without the fix.

habdelra · 2026-08-29T21:14:27Z

[Claude Code 🤖] Confirmed manually in code mode now that I could drive a signed-in browser: New+ → Card Instance → General Card creates the instance, the preview opens straight into edit format with its fields live, typing into Name lands immediately and the editor reports Saved, and the value is on the realm's file system. No placeholder at any point, before or after indexing.

The card a client waits on is one another client created: the bytes land on the realm, this tab hears about it through the realm's file event, and nothing of the card has passed through its store — which is what makes the placeholder the right thing to show rather than a stand-in for a card it already holds. The test now takes that route and asserts the store is empty of it before the visit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

lukemelia

[Claude Code 🤖] This review traced the awaiting-index marker end to end — the realm's two 404 branches and the error serialization, the store paths that install and clear the placeholder, every host surface that branches on a 404 card error, and the races between the marker and the index events that resolve it.

Bottom line: no blocking issues. One gap in the new event machinery is worth closing in this PR (inline comment), and the description's host-mode claim doesn't hold.

A bare full index event that lands while a card's first read is in flight can still strand the placeholder — see the inline comment on reloadAwaitingIndexInstances in packages/host/app/services/store.ts.
Host mode does not pick the placeholder up, though the description says it does. isNotFound in packages/host/app/components/host-mode/card.gts branches on status === 404 before CardError ever renders, so an awaiting-index 404 gets the hard "This page could not be found" screen. And wiring it is not the one-line guard it looks like: subscribeToRealm returns early when hostModeService.isActive, so host mode receives no index events and a placeholder there would never resolve. Either scope the description to the surfaces that actually reach the CardError branch, or file host-mode support (placeholder plus some resolution path) as a concrete follow-up.
A question on whether this closes the deployed symptom: on a multi-replica deployment on shared NFS storage, a peer replica can serve the read before the freshly written file is visible to it at all (stale negative dentry cache). That replica takes the !source early-out in missingInstanceResponse and returns the unmarked 404, so the error banner still appears for exactly the cross-replica case the description calls out. Worth verifying against a deployed environment that the observed window is index-lag rather than file-visibility before treating the symptom as closed.

Adjacent, out of scope: packages/host/app/templates/index.gts composes a "Card not found" document title from any 404 error, awaiting-index included — cosmetic, and its fate follows whatever item 2 decides.

lukemelia · 2026-08-31T02:06:37Z

+  // Re-read every card being held as awaiting-index in `realmURL`. Their whole
+  // state is "a row for me is coming", and a from-scratch pass is one way it
+  // arrives without any event naming the card.
+  private reloadAwaitingIndexInstances(realmURL: string): number {


[Claude Code 🤖] A bare full event that arrives while a card's first read is still in flight strands the placeholder: this sweep reads cardErrorEntries(), and the in-flight read has recorded nothing yet — the awaiting-index 404 it settles into is installed after this event was handled, and no later event is coming for it. It is the same race the incremental branch closes with hasInflightCardLoad, and the machinery already exists; the full branch just doesn't use it:

for (let id of this.inflightGetCards.keys()) { if (id.startsWith(realmURL)) { this.reloadAfterInflightLoad.perform(id); } }

(reloadAfterInflightLoad already re-checks peekError after the read settles, so a read that comes back with the real card is left alone.)

Regression class — a gap in machinery this change introduces — and non-blocking: the window is as narrow as the incremental one, but the startup reindex that emits the bare full event is exactly when early reads are most likely.

[Claude Code 🤖] You're right, and it is the same race — fixed in 90991a6. The sweep now walks inflightGetCards for the realm alongside the recorded placeholders and hands each one to reloadAfterInflightLoad.

One deviation from your sketch: the realm test is this.realm.realmOf(rri(id)) !== realmURL rather than id.startsWith(realmURL), matching what the recorded-placeholder loop just above already does. A realm key is whatever form the realm was registered under, so a prefix match would miss an id in a mapped or aliased form.

packages/host/tests/integration/store-test.gts covers it: the read is parked on its 404, the realm indexes the file, a bare full event is delivered inside that window, and the read is then released. Without the new loop the placeholder is stranded and both assertions fail; with it the card takes over.

The sweep a bare `full` event triggers reads the placeholders the store has recorded, which a read still in flight has not produced yet — the awaiting-index error it settles into is installed after the event was handled, and nothing further is coming to say the row now exists. The in-flight reads for the realm get the same deferred reload an invalidation naming a card mid-read already gets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

habdelra · 2026-08-31T14:25:04Z

[Claude Code 🤖] Thanks — all three land. Taking them in turn:

1. The full-index sweep's in-flight gap — fixed in 90991a6, replied inline.

2. Host mode — you're right on both halves, and the description was wrong. isNotFound branches on status === 404 at host-mode/card.gts:51 and returns the "This page could not be found" screen before CardError is reached, and subscribeToRealm returns early when hostModeService.isActive, so a placeholder there would have nothing to resolve it. I have scoped the description to the surfaces that actually reach the branch and said why host mode is not among them, rather than implying it is a one-line guard away.

3. Cross-replica file visibility — a fair challenge, and I cannot settle it from here. You are right about the behaviour: a replica that cannot yet see the file takes the !source early-out and returns the unmarked 404, so the marker is best-effort by construction — it reports what the serving replica can observe, and a replica that cannot see the file is not in a position to promise the card is coming. What I have verified is the index-lag window, locally and deterministically; that the deployed symptom is that window rather than file visibility is an assumption I have not tested against a deployed environment. Worth confirming before treating the reported symptom as closed, and I would rather that be recorded as open than implied shut.

On the adjacent templates/index.gts "Card not found" title: agreed it is cosmetic and that its fate follows the host-mode decision, so I have left it.

habdelra and others added 3 commits August 28, 2026 12:01

Add PR preview screenshots

63b1722

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Remove PR preview screenshots

15e5b1f

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector Bot reviewed Aug 28, 2026

View reviewed changes

Comment thread packages/runtime-common/realm.ts Outdated

Comment thread packages/runtime-common/realm.ts Outdated

habdelra and others added 4 commits August 28, 2026 12:30

habdelra requested a review from a team August 28, 2026 23:13

lukemelia reviewed Aug 31, 2026

View reviewed changes

lukemelia approved these changes Aug 31, 2026

View reviewed changes

habdelra merged commit 069a353 into main Aug 31, 2026
70 of 71 checks passed

Navigation Menu

Uh oh!

Tell a card awaiting its first index pass apart from one that is missing - #5929

Tell a card awaiting its first index pass apart from one that is missing#5929
habdelra merged 10 commits into
mainfrom
cs-11074-after-uploading-file-it-often-shows-a-broken-link

habdelra commented Aug 28, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

habdelra commented Aug 29, 2026

Uh oh!

habdelra commented Aug 29, 2026

Uh oh!

habdelra commented Aug 29, 2026

Uh oh!

lukemelia left a comment

Uh oh!

lukemelia Aug 31, 2026

Uh oh!

habdelra Aug 31, 2026

Uh oh!

habdelra commented Aug 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

habdelra commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before / after

Testing

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preview deployments

Host Test Results

Realm Server Test Results

Uh oh!

habdelra commented Aug 29, 2026

Uh oh!

habdelra commented Aug 29, 2026

Uh oh!

habdelra commented Aug 29, 2026

Uh oh!

lukemelia left a comment

Choose a reason for hiding this comment

Uh oh!

lukemelia Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

habdelra Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

habdelra commented Aug 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

habdelra commented Aug 28, 2026 •

edited

Loading

github-actions Bot commented Aug 28, 2026 •

edited

Loading